Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review

There is an increasing demand for using Unmanned Aerial Vehicle (UAV), known as drones, in different applications such as packages delivery, traffic monitoring, search and rescue operations, and military combat engagements. In all of these applications, the UAV is used to navigate the environment autonomously --- without human interaction, perform specific tasks and avoid obstacles. Autonomous UAV navigation is commonly accomplished using Reinforcement Learning (RL), where agents act as experts in a domain to navigate the environment while avoiding obstacles. Understanding the navigation environment and algorithmic limitations plays an essential role in choosing the appropriate RL algorithm to solve the navigation problem effectively. Consequently, this study first identifies the main UAV navigation tasks and discusses navigation frameworks and simulation software. Next, RL algorithms are classified and discussed based on the environment, algorithm characteristics, abilities, and applications in different UAV navigation problems, which will help the practitioners and researchers select the appropriate RL algorithms for their UAV navigation use cases. Moreover, identified gaps and opportunities will drive UAV navigation research

Reinforcement Learning Algorithms: An Overview and Classification

The desire to make applications and machines more intelligent and the aspiration to enable their operation without human interaction have been driving innovations in neural networks, deep learning, and other machine learning techniques. Although reinforcement learning has been primarily used in video games, recent advancements and the development of diverse and powerful reinforcement algorithms have enabled the reinforcement learning community to move from playing video games to solving complex real-life problems in autonomous systems such as self-driving cars, delivery drones, and automated robotics. Understanding the environment of an application and the algorithms’ limitations plays a vital role in selecting the appropriate reinforcement learning algorithm that successfully solves the problem on hand in an efficient manner. Consequently, in this study, we identify three main environment types and classify reinforcement learning algorithms according to those environment types. Moreover, within each category, we identify relationships between algorithms. The overview of each algorithm provides insight into the algorithms’ foundations and reviews similarities and differences among algorithms. This study provides a perspective on the field and helps practitioners and researchers to select the appropriate algorithm for their use case

Virtual Sensor Middleware: A Middleware for Managing IoT Data for the Fog-Cloud Platform

Internet of Things is a massively growing field where billions of devices are connected to the Internet using different protocols and produce an enormous amount of data. The produced data is consumed and processed by different applications to make operations more efficient. Application development is challenging, especially when applications access sensor data since IoT devices use different communication protocols. The existing IoT architectures address some of these challenges. This thesis proposes an IoT Middleware that provides applications with the abstraction required of IoT devices while distributing the processing of sensor data to provide a real-time or near real-time response and enable the applications to choose from where to consume sensor data. The suggested middleware architecture minimizes the development efforts required by the applications by automating the processing of sensor data on multiple nodes (fog nodes) deployed near IoT devices and making it configurable. Furthermore, the dissemination of sensor data using the publish-subscribe paradigm makes it easier for applications to decide from where to consume sensor data while maintaining decoupling from IoT devices

Agglomerative Hierarchical Clustering with Dynamic Time Warping for Household Load Curve Clustering

Energy companies often implement various demand response (DR) programs to better match electricity demand and supply by offering the consumers incentives to reduce their demand during critical periods. Classifying clients according to their consumption patterns enables targeting specific groups of consumers for DR. Traditional clustering algorithms use standard distance measurement to find the distance between two points. The results produced by clustering algorithms such as K-means, K-medoids, and Gaussian Mixture Models depend on the clustering parameters or initial clusters. In contrast, our methodology uses a shape-based approach that combines Agglomerative Hierarchical Clustering (AHC) with Dynamic Time Warping (DTW) to classify residential households\u27 daily load curves based on their consumption patterns. While DTW seeks the optimal alignment between two load curves, AHC provides a realistic initial clusters center. In this paper, we compare the results with other clustering algorithms such as K-means, K-medoids, and GMM using different distance measures, and we show that AHC using DTW outperformed other clustering algorithms and needed fewer clusters

Virtual Sensor Middleware: Managing IoT Data for the Fog-Cloud Platform

This paper introduces the Virtual Sensor Middleware (VSM), which facilitates distributed sensor data processing on multiple fog nodes. VSM uses a Virtual Sensor as the core component of the middleware. The virtual sensor concept is redesigned to support functionality beyond sensor/device virtualization, such as deploying a set of virtual sensors to represent an IoT application and distributed sensor data processing across multiple fog nodes. Furthermore, the virtual sensor deals with the heterogeneous nature of IoT devices and the various communication protocols using different adapters to communicate with the IoT devices and the underlying protocol. VSM uses the publish-subscribe design pattern to allow virtual sensors to receive data from other virtual sensors for seamless sensor data consumption without tight integration among virtual sensors, which reduces application development efforts. Furthermore, VSM enhances the design of virtual sensors with additional components that support sharing of data in dynamic environments where data receivers may change over time, data aggregation is required, and dealing with missing data is essential for the applications

Deep Reinforcement Learning for Autonomous Unmanned Aerial Vehicle Navigation

Unmanned Aerial Vehicles (UAVs) are instrumental in various tasks, including package delivery, disaster response, and surveillance. Their varied applications highlight the need for advanced navigation techniques, with Deep Reinforcement Learning (DRL) being a key approach in enhancing UAV autonomy. The challenges in UAV navigation using DRL span three key areas: comprehending DRL applications in UAV navigation, navigation frameworks accommodating the requirements of autonomous UAV navigation, and adaptive DRL algorithms handling high-dimensional inputs and temporal dependencies inherent in UAV navigation. In response to these challenges, this thesis explores challenges associated with DRL in autonomous UAV navigation in complex 3D environments. The investigation accentuates understanding algorithmic properties and navigation tasks to leverage DRL methodologies in UAV navigation. The DRL algorithms for autonomous UAV navigation are investigated and classified. The comprehensive review includes over fifty Reinforcement Learning (RL) algorithms, their traits, relations, and classifications based on the application environment and UAV navigation. Moreover, a process for selecting the appropriate DRL algorithm based on the navigation environment and algorithmic needs is presented. Next, the thesis presents VizNav, a modular RL-based navigation framework that addresses the current challenges in RL-based autonomous UAV navigation, leveraging off-policy RL algorithm and employing Prioritized Experience Replay (PER) for improved UAV navigation results and algorithm convergence. Additionally, VizNav uses Depth Map Images (DMI) to provide the agent with a more accurate and comprehensive depth perspective, enhancing UAV navigation. VizNav experimental results reveal enhanced navigation using TD3 supported by PER and DMI while maintaining adaptability using different algorithms and environments. Finally, this thesis proposes Agile Deep Q-Network (AG-DQN), a novel DRL algorithm to manage high-dimensional inputs and temporal dependencies, employing a dynamic multi-glimpse strategy and advanced temporal processing to selectively and dynamically extract salient features for improved decision-making. AG-DQN outperforms other state-of-the-art methods like DRQN and DARQN in complex UAV navigation tasks, using only 32% of the total image pixels (environment state). Overall, the thesis contributes to developing fully autonomous UAVs capable of navigating various scenarios, paving the way for their broadened applications